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MDRC is pleased to have this opportunity to provide additional information for the 
consideration of Chairman Alexander, Ranking Member Murray, and members of the Committee 
on ways research can be used to improve the academic success of low-income college students. 

MDRC — a nonprofit, nonpartisan research organization based in New York City and 
Oakland, California — was founded more than 40 years ago to build reliable evidence on the 
effectiveness of programs for the disadvantaged and to help policymakers and practitioners use 
that evidence to improve policies and programs. MDRC is kn own for conducting large-scale 
evaluations and demonstration projects to test the impacts and cost-effectiveness of education 
and social programs. Many MDRC studies use a random assignment research design, the most 
rigorous method for evaluating such programs, which is able to determine the value an 
intervention adds to the status quo. This method, analogous to the one used in medical clinical 
trials, produces the most reliable evidence that a program works. As a result, it is the primary 
method to be accepted without reservations by the U.S. Department of Education’s What Works 
Clearinghouse (WWC). 

The goal of this additional testimony is to reiterate the second recommendation in the 
testimony of Dr. Lashawn Richburg-Hayes to “encourage innovation paired with research, 
especially rigorous evaluation.” The testimony of Senators and witnesses at the August 5, 2015 
hearing provided a number of promising ideas that are operationally feasible and whose 
outcomes seem encouraging, but that have not yet been subjected to rigorous evaluation based on 
WWC standards. In addition, other evaluated programs that were mentioned produced small 
effects at best on outcomes identified by the Institute of Education Sciences (IES) as key markers 
of collegiate progress. In a time of limited resources, building reliable evidence before making 
major investments is essential to the long-term goal of increasing college completion. 

This additional testimony is intended to emphasize the importance of building into the 
reauthorization of the Higher Education Act incentives for innovation, coupled with 
requirements for rigorous evaluation through randomized experiments whenever feasible. This 
testimony identifies opportunities in two areas — financial aid and student support services — 
and offers caution in the area of performance funding for higher education. 

A summary of our recommendations follows: 

1. Test variations of enrollment intensity tied to Pell Grant receipt before altering the 
policy. The Department of Education can test Pell Grant funding to cover the summer 
tenn of the academic year. Offering Pell Grant aid to students during the summer would 
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offer an opportunity to test whether aid during short summer or winter terms (that is, 
those less than 12 weeks in duration) helps students make stronger progress toward 
degree completion. Such tests could evaluate tying the reintroduction of summer Pell 
awards to some of the other strategies (for example, incremental aid disbursements) that 
could help control program costs and make the program more sustainable. Tests (some 
taking as little time as a semester to gain pertinent infonnation) could be designed to 
evaluate the effectiveness of using the Pell Grant as an incentive to enroll for 15 credits 
per semester, offering the Pell Grant for reaching particular milestones, and the relative 
effectiveness of changes to the Pell Grant versus alternative funding for students on the 
verge of dropping out due to tuition and fees. 

2. Include student support services as an IES grant priority and advertise this priority 
clearly through the Federal Register and other means before competitions open. 

Congress can capitalize on the IES annual grant opportunity to provide more information 
in areas of interest to policymakers, the Department of Education, and other stakeholders. 
One way to implement this is to identify specific competitions to focus proposal 
submissions in a desired area. Announcing the priority broadly would allow colleges and 
researchers to prepare models and find partners. 

3. Continue to use First in the World (FITW) to encourage innovation and research in 
student support services. The federal government has made notable strides in valuing 
evidence through the Investing in Innovation (i3) program and FITW. Both grant 
opportunities support programs while requiring strong evaluation that will benefit 
policymakers and practitioners. In addition, both use a tiered strategy of financial 
incentives, providing the most funding to expand proven strategies to a large scale and 
lesser amounts to support innovation and a learning agenda. This year’s FITW 
competition included “Improving Student Support Services” as an absolute priority in the 
validation grant category, further directing attention to areas where the research base is 
thin. Future competitions could foster additional innovation and research on support 
services. 

4. Consider the unintended consequences of implementing performance funding in 
higher education in the absence of standardized metrics of institutional 
effectiveness. While performance funding could provide incentives for an increase in 
institutional effectiveness, it could also lead institutions to select students who are more 
likely to graduate, to lower their institutional standards to achieve desired outcomes, or 
both. While there are strong critiques to be made of all funding schemes, it is likely that 
the development of standard metrics should precede the implementation of a 
performance-funding alternative. 

Financial Aid Reforms 

Limited financial resources and the related need to work for pay may be major barriers to 
academic success for many low-income college students. Financial aid can help alleviate these 
barriers, but significant challenges remain for many students. MDRC believes that innovations 
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to financial aid disbursement can contribute to improving academic outcomes. A number of 
financial aid refonns studied by MDRC as well as others were mentioned during the hearing: 3 


• Using the Pell Grant as an incentive to enroll for 15 credits per tenn 

• Concern about the unintended consequences of tightening financial aid requirements by, 
for example, requiring 1 5 credits 

• Using incentive-based grants to align student behavior with desired outcomes 

• Offering Pell Grants for reaching particular milestones 

• Simplifying the Free Application for Federal Student Aid (FAFSA) 

• Implementing year-round Pell Grants 

• Experimenting with when and how students get aid 

• Providing a one-time micro-grant to students on the verge of dropping out due to tuition 
and fees (as in the Panther Grant program at the Georgia State University) 

• Offering more financial aid 

• Providing loan forgiveness as an incentive to graduate 

A common theme of all of these ideas is changing the incentive structure of the Pell 
Grant or other forms of financial aid to motivate a change in student behavior that will result in 
sustained enrollment and eventual completion. Items on this list could be viewed as providing 
multiple ways to accomplish the same end (for example, a 15-credit requirement for Pell Grants, 
year-round Pell Grants, one-time micro-grants, and Pell Grants for milestones could be various 
ways to encourage students to attempt a larger number of credits). Some ideas involve ways to 
use existing aid more intentionally to motivate behavior (for example, experimenting with how 
and when students get aid or employing an incentive-based aid design in existing institutional 
grant models or in implementing loan forgiveness). 

There are also many other alternatives with potential to achieve the desired outcomes. For 
example, a public service campaign such as Hawaii’s “15 to Finish” could improve the 
proportion of students attempting 15 credits. 4 Creating student schedules with 15 credits being 
the default could have a similar effect. 

Since it is unclear which approaches are most effective, studies can be designed to look at 
the efficacy of these alternatives and to explore any unintended consequences before an 
extensive policy change is made. One idea is to recruit colleges to volunteer to be randomly 
assigned to various policy alternatives. School-based randomized controlled trials have been 
employed in K-12 education in many instances, and several organizations have the ability to 
administer such a design. A similar idea would be to use the experimental sites — authorized by 
Congress under section 487A(b) of the Higher Education Act of 1965, as amended — to recruit 
colleges and evaluate the interventions through randomized trials. 

Student Support Services 

Several of the Senators and witnesses highlighted student support services — such as 
academic advising, student success centers, and technology-based support service — as 
promising strategies to improve student success. There is some research on the impact of these 
types of interventions, but few interventions have produced clear evidence that they are 
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beneficial, and more could be done to encourage the building of knowledge regarding effective 
practices in this area. 

Some studies of academic advising have been done, although there is still much to learn 
about effective practices. Academic advising is intended to help students address such topics as 
choosing which classes to take, balancing school with other responsibilities, interacting with 
professors, and staying on track to graduate on time. MDRC conducted an experimental 
evaluation of an enhanced academic advising program and found that the program modestly 
increased students’ persistence and the number of credits students earned in the short term. 5 The 
City University of New York’s Accelerated Study in Associate Programs (ASAP) also provided 
intensive advising to students, sustained over three years, and produced dramatic impacts, but it 
was combined with multiple other program components, so the impact of the advising 
component alone is not known. A recent experimental evaluation of an individually tailored 
student coaching program provided by InsideTrack is an important exception. The evaluation 
found positive effects on persistence at least one year after the coaching had ended. 6 

One MDRC evaluation found that a student success course had modest, short-term 
impacts on credits earned and students’ academic standing, while another found positive effects 
on students’ self-management, self-awareness, and engagement in college, but few overall 
effects on students’ academic achievement. 7 Student success programs are popular, but vary 
across colleges. They may be an effective way to help students, but the evidence is still thin. 

The potential of technology-based support services — often called Integrated Planning 
and Advising Services (IPAS) — to improve student outcomes is still largely untested. IPAS 
software typically provides students with advising and degree planning and uses predictive 
analytics and early alerts. Little rigorous research, however, has been conducted to date on the 
impact of these technology-based solutions. 8 The Bill and Melinda Gates Foundation is 
supporting the development of IPAS software in partnership with various colleges, which will 
add infonnation to the field. 9 MDRC is also conducting a random assignment study of a web- 
based, personalized advising tool, which aims to help students create counselor-approved 
academic plans. 10 

Some Cautions to Consider About Performance Funding in Higher Education 

The potential of performance funding for higher education institutions was an important 
focus of the August 5 hearing. While MDRC acknowledges that rigorous evaluation in this area 
is unlikely, it seems important to carefully consider the possible unintended consequences of 
such policies. Most importantly, performance funding is likely to punish many effective 
institutions and reward many ineffective institutions, potentially undermining its intentions. 

There are at least three broad ways for colleges to improve institutional outcomes (for 
example, graduation rates): 

(1) Improve institutional effectiveness 

(2) Select students who are more likely to graduate 

(3) Lower institutional standards to achieve desired outcomes 
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The main goal of performance funding should be to provide incentives for the first 
strategy while avoiding the other two. The typical solution suggested to protect against colleges 
using the second strategy is to use input-adjusted outcomes (where outcomes are adjusted to 
account for differences in student demographics), and the typical solution suggested to protect 
against the third is to rely on accreditors. While these solutions may mitigate concerns, they will 
not eliminate them, and it is unclear whether they will diminish them significantly. 

Two major obstacles to comparing colleges’ relative effectiveness are: (1) institutions 
serve different types of students, and (2) generally, there are not standardized outcomes in higher 
education. As discussed during the hearing, there are ways to mitigate (although not eliminate) 
concerns about the first obstacle. For example, institutions could receive bonus points for serving 
low-income students whose likelihood of succeeding is, on average, lower than their higher- 
income counterparts. 1 1 Making adjustments like this may make cross-institutional comparisons 
fairer, but such approaches will not result in apples-to-apples comparisons. For example, the 
low-income students attending open-access four- year colleges are not the same as the low- 
income students attending elite four-year colleges. Making such adjustments based on measured 
characteristics will only partially address the fact that institutions serve different types of 
students with different propensities to succeed. 

The even bigger challenge is that proposed outcomes for perfonnance funding are not 
standardized, making cross-institutional comparisons ambiguous at best, and meaningless at 
worst. For example, consider a common performance-funding outcome like degree completion. 
Little is known about how much variation exists across institutions in the difficulty of earning a 
degree. One reason for this is that we cannot disentangle difficulty of earning a degree from 
institutional effectiveness. To make the point clear, even if College X and College Y both serve 
students who look identical upon their entrance, comparing their graduation rates does not enable 
one to know whether (a) one college is more effective than the other or (b) one college simply 
has higher standards than the other. 

While it may be impossible to quantify how much variation in difficulty of earning a 
degree there is in higher education, research in K-12 education can provide some guidance. An 
analogous situation has been carefully examined in K-12 schools, where states are required to 
report the percentages of students achieving proficiency in reading and mathematics on statewide 
exams. Importantly, each state administers a unique state exam with unique content and a unique 
proficiency cut score (much as each college can be thought of as having its own unique 
requirements to graduate, and its own unique stringency to meet those graduation requirements). 

Because the state exams are different, one might wonder whether there is utility in 
making cross-state comparisons in proficiency rates on these state exams, in order to determine 
which states have the highest-achieving students. The U.S. Department of Education 
commissioned a paper to examine this issue taking advantage of the National Assessment of 
Educational Progress (NAEP), a national standardized exam that allows for valid cross-state 
comparisons. “ The general conclusion of the study is that: “The observed heterogeneity in 
states ’ reported percent proficient can be largely attributed to differences in the stringency of 
their standards. ” n 
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In other words, the study found that if you compare the percentage proficient in one state 
with the percentage proficient in another state using an outcome that is not standardized across 
states, differences largely have to do with the stringency of standards. Simply put: It may be 
invalid to use nonstandardized outcomes, like college graduation rates, to make cross- 
institutional comparisons. 

Below is a plot of the data from that U.S. Department of Education-commissioned paper. 
On the x-axis is each state in the United States. The y-axis shows the percentage of fourth- 
graders meeting proficiency standards using their state’s math test (the leftmost bar, in gray) and 
the NAEP (the rightmost bar, in black). Although each state’s test is designed to measure 
mathematics achievement, the tests are different and the proficiency definitions are different. 
NAEP also measures mathematics achievement, but unlike the state exams, all students in the 
country take the same test using the same definition of proficiency. There is almost no 
relationship between the percentage proficient on NAEP and the percentage proficient using the 
state test. Massachusetts exemplifies the problem: Looking at the NAEP scores, Massachusetts 
has the highest-achieving fourth- graders in the country. However, because Massachusetts’ state 
test is extremely difficult, the state ranks fourth from the bottom on the percentage of students 
passing their own statewide exam. If K-12 education implemented perfonnance funding based 
on the percentage proficient on state exams, Massachusetts — the state with the highest- 
achieving students — would be penalized. 
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This example illustrates that rewarding or punishing colleges based on their relative 
effectiveness is going to be problematic. Perfonnance funding could penalize those operating in 
the desired direction while rewarding others who may not merit the resources, as no standard 
exists to adequately measure relative performance. 

Recommendations 

1. Test variations of enrollment intensity tied to Pell Grant receipt before altering the 
policy. The Department of Education can test Pell Grant funding to cover the summer 
term of the academic year. Offering Pell Grant aid to students during the summer would 
offer an opportunity to test whether aid during short summer or winter terms (that is, 
those less than 12 weeks in duration) helps students make stronger progress toward 
degree completion. Such tests could evaluate tying the reintroduction of summer Pell 
awards to some of the other strategies (for example, incremental aid disbursements) that 
could help control program costs and make the program more sustainable. In addition, 
tests (some taking as little time as a semester to gain pertinent infonnation) could be 
designed to evaluate the effectiveness of using the Pell Grant as an incentive to enroll in 
1 5 credits, offering the Pell Grant for reaching particular milestones, and the relative 
effectiveness of changes to the Pell Grant versus alternative funding for students on the 
verge of dropping out due to tuition and fees. 

2. Include student support services as an IES grant priority and advertise this priority 
clearly through the Federal Register and other means before competitions open. 

Congress can capitalize on the IES annual grant opportunity to provide more infonnation 
in areas of interest to policymakers, the Department of Education, and other stakeholders. 
One way to implement this is to identify specific competitions to focus proposal 
submissions in a desired area. Announcing the priority broadly would allow colleges and 
researchers to prepare models and find partners. 

3. Continue to use First in the World (FITW) to encourage innovation and research in 
student support services. The federal government has made notable strides in valuing 
evidence through the Investing in Innovation (i3) program and FITW. Both grant 
opportunities support programs while requiring strong evaluation that will benefit 
policymakers and practitioners. In addition, each uses a tiered strategy of financial 
incentives, providing the most funding to expand proven strategies to a larger scale and 
lesser amounts to support innovation and a learning agenda. This year’s FITW 
competition included “Improving Student Support Services” as an absolute priority in the 
validation grant category, further directing attention to areas where the research base is 
thin. Future competitions could foster additional innovation and research on support 
services. 

4. Consider the unintended consequences of implementing performance funding in 
higher education in the absence of standardized metrics of institutional 
effectiveness. While performance funding could provide incentives for an increase in 
institutional effectiveness, it could also lead institutions to select students who are more 
likely to graduate, to lower their institutional standards to achieve desired outcomes, or 
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both. A large challenge in performance funding is the meaningfulness of outcomes like 
degree completion as a measure of schools’ relative effectiveness (even if those outcomes 
are adjusted for the proportion of low-income students served, for example). Little is 
known about how much variation exists across institutions in the difficulty of earning a 
degree because administrators and researchers cannot disentangle the difficulty of earning 
a degree from institutional effectiveness. As a result, there is no meaningful way to define 
effectiveness. While there are strong critiques to be made of all funding schemes, it is 
likely that the development of standard metrics should precede the implementation of a 
performance-funding alternative. 
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